Research Report: Automated Visualization

1. Executive summary & goals

This report examines publication and audience engagement trends for Towards Data Science (TDS) articles from the first recorded month in 2010 through mid-2021. It is written for visualization researchers, data‑science students, and Medium/TDS contributors who want a clear picture of how publishing volume, author contributions, and reader engagement (claps, responses, and reading time) have evolved. Our goals are to describe the dataset and cleaning steps, characterize temporal publishing patterns and the rise of paid content, quantify engagement and reading-efficiency patterns, identify highly engaged articles and influential authors, and surface practical recommendations for authors and automated analytics dashboards. Key questions include: how has publication volume and paid-content adoption changed over time, how concentrated is audience attention, how do paid and free articles compare on engagement, and what reading-time windows are associated with stronger audience response.

2. Data, cleaning, and derived metrics

The core dataset contains 46,079 article records (no duplicates after deduplication) with fields including publish_date, author, title, claps, responses, reading_time (minutes), and a normalized paid flag. Cleaning steps included parsing datetimes, converting text blanks to empty strings, treating NaN claps/responses as 0, normalizing paid to a boolean, and removing one row with non-positive reading_time when computing reading-efficiency. Derived features used throughout the report are: engagement_score (claps + responses, used as the primary combined engagement metric), average claps per response (claps divided by max(1,responses)), reading_efficiency (claps per minute = claps / reading_time), monthly counts and paid_ratio (proportion of articles marked paid for each month). Summary diagnostics show heavy skew and long tails: claps mean 266.3 (median 87), responses mean 1.71 (median 0), reading_time mean 7.17 min (median 6), engagement_score mean 283.4 (median 94), and reading_efficiency mean 40.94 claps/min (median 13.5). Outliers are present above the 99th percentiles (e.g., claps > 3100, engagement > 3330) and include a small set of viral articles (top engagement ~54,980) that drive most extreme values; zero-clap articles are rare (~0.7%). These checks confirm strong right skew, a small number of extreme high-engagement articles, and overall data consistency after cleaning.

The visual panel of distributions and monthly summaries highlights two consistent facts: engagement and related metrics are strongly right‑skewed, and publication volume plus paid-content share have increased sharply over time. Log-scaled histograms show that most articles cluster at low-to-moderate clap and engagement values with a long tail of viral posts (the 99th percentile thresholds are ~3100 claps and ~3330 engagement). The monthly time-series view shows a slow start through 2015, a moderate growth phase around 2016–2019, and a major surge from 2020 onward; monthly totals peak in mid‑2020 (May–July 2020 had the largest counts, e.g., May 2020 = 2,208 articles). The paid_ratio line rises with volume: overall 64.6% of articles are paid, and recent months remain high (recent 6‑month average ~63.8%), indicating that paid content became the dominant mode during the 2020+ expansion. Taken together, these visuals validate the data (expected heavy tails and a clear temporal shift toward both higher volume and higher paid-ratio) and point to a small number of high‑engagement outliers that should be treated separately when modeling central tendencies.

3. Temporal publication trends (time-series)

The temporal analysis focuses on how publishing activity evolved and how paid-content adoption changed alongside growth. We segment the timeline into three phases: pre‑2015 (very low volume), 2016–2019 (steady growth and experimentation), and 2020–present (rapid expansion and heavy paid-content adoption). Monthly seasonality is visible, with consistently strong calendar months (May, July, and September), and several clear peak months in 2020. The analysis aims to quantify growth rates, highlight seasonal patterns, and show how the paid_ratio moved from a minority share to the majority of articles during the expansion.

The time‑series visualization pairs stacked (paid vs free) yearly bars with a smoothed monthly stream that makes the growth phases easy to see. Yearly bars show a modest paid share early on and an increasing contribution from paid articles each year; monthly smoothing (3‑month rolling) shows slow pre‑2015 output, a moderate 2016–2019 phase (about 17,900 articles with paid_ratio ≈ 46.6% and avg monthly ≈ 459), and a dramatic 2020+ phase (about 28,142 articles with paid_ratio ≈ 76.1% and avg monthly ≈ 1,481). Peak volume occurred in May–July 2020 (May 2020 = 2,208 total, of which ~1,870 were paid), and seasonality places high averages in September, May, and July. The smoothed end-of-series value is about 1,138 articles (3‑month smoothed), confirming sustained elevated activity. In short, the visualization makes two points clear: publication volume rose sharply in 2020 and paid content became the dominant format during that surge.

4. Engagement patterns and author-level analysis

Audience engagement is highly concentrated: distributions of claps and the combined engagement_score are long-tailed with a small fraction of viral articles accounting for extremely high values. The median engagement (94) is far below the 99th percentile (~3,330), so central tendencies understate the role of outliers. Authors are unevenly productive and influential — the top authors by total engagement include names like Will Koehrsen (≈376k total engagement across 105 articles) and George Seif (≈180k across 90 articles) — but author lists show different leaders by article count versus total engagement. Comparing paid and free articles reveals a notable difference: free articles (16,319 records) have higher mean engagement (≈368.7) and a larger share of the top 1% engagement articles (top‑1% fraction ≈ 1.7% for free vs 0.6% for paid), despite paid articles being more numerous (29,760). Correlation patterns and reading efficiency measures (claps per minute) suggest that longer reading times modestly relate to higher engagement but that most of the engagement signal is driven by a small set of highly successful articles.

The engagement visualization juxtaposes log‑scale distributions and author-level totals to emphasize skew and concentration. The log10(1 + value) histograms show that most articles have low-to-moderate claps and engagement while a long right tail captures viral pieces; summary log statistics (mean log10(1+claps) ≈ 1.91, mean log10(1+engagement) ≈ 1.95) reflect this compression. A ranked bar view of top authors by total engagement highlights that a small number of authors accumulate outsized engagement (e.g., top author ~376k engagement). Thresholds for the top 1% identify ~3,330 engagement as the cutoff, and free articles are disproportionately represented among these top performers. These visuals support the takeaway that engagement is not uniformly distributed across articles or authors: attention is concentrated and a minority of pieces and contributors drive most measurable impact.

5. Conclusions, recommendations, and visualization benchmark

Key takeaways and practical recommendations: publishing on TDS became far more active after 2020 and is now dominated by paid content, but free articles still tend to capture a disproportionate share of top engagement. Engagement is heavily right‑skewed—median and mean differ substantially—so authors and dashboards should report both central and tail metrics. For authors, aim for concise but substantive reading times in the mid range (roughly 7–12 minutes) where median and 90th‑percentile engagement are strong; longer pieces (10+ minutes) can still perform well, especially for paid content, but the incremental gain is modest and attention remains concentrated in a few viral posts. For strategy: free articles have higher mean engagement and greater representation in the top 1% of posts, while paid articles are more common and show slightly lower median engagement, so authors should balance monetization against discoverability and shareability. For analysts and dashboards, include both distributional visualizations (log histogram of claps/engagement), temporal trends (monthly counts, paid_ratio with rolling averages), author leaderboards (by count and by total engagement), engagement per reading_time bin, and outlier-aware KPIs (median, 90th/99th percentiles, top‑1% threshold = 3,330 engagement).

The recommended benchmark visualization compares median and 90th‑percentile engagement across reading_time bins for paid versus free articles. The aggregated view shows that median engagement increases with reading time for both groups (e.g., paid median rises from ≈58.5 for 1–3 min to ≈120 for 10+ min; free median rises from ≈54 to ≈149 across the same bins), while 90th‑percentile engagement highlights the tail: free articles reach higher 90th values (e.g., p90 up to ≈1,340 in 10+ min) and a higher chance of reaching top-tier engagement. Correlation tests show a weak positive association between reading_time and engagement (Spearman ≈0.181 for free, ≈0.136 for paid), indicating reading time is one of several modest predictors of engagement. This chart supports a practical rule: targeting the 7–12 minute reading window often balances depth and audience attention, and dashboards should present both medians and tail percentiles to capture typical and exceptional performance.